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ABSTRACT 

The  U.S.  Army  Research  Laboratory  (ARL)  has  been  investigating  the  utility  of  ultra- wideband  (UWB)  synthetic 
aperture  radar  (SAR)  technology  for  detecting  concealed  targets  in  various  applications.  We  have  designed  and  built  a 
vehicle-based,  low-frequency  UWB  SAR  radar  for  proof-of-concept  demonstration  in  detecting  obstacles  for 
autonomous  navigation,  detecting  concealed  targets  (mines,  etc.),  and  mapping  internal  building  structures  to  locate 
enemy  activity.  Although  the  low-frequency  UWB  radar  technology  offers  valuable  information  to  complement  other 
technologies  due  to  its  penetration  capability,  it  is  very  difficult  to  comprehend  the  radar  imagery  and  correlate  the 
detection  list  from  the  radar  with  the  objects  in  the  real  world. 

Using  augmented  reality  (AR)  technology,  we  can  superimpose  the  information  from  the  radar  onto  the  video 
image  of  the  real  world  in  real-time.  Using  this,  Soldiers  would  view  the  environment  and  the  superimposed  graphics 
(SAR  imagery,  detection  locations,  digital  map,  etc.)  via  a  standard  display  or  a  head-mounted  display.  The 
superimposed  information  would  be  constantly  changed  and  adjusted  for  every  perspective  and  movement  of  the  user. 
ARL  has  been  collaborating  with  ITT  Industries  to  implement  an  AR  system  that  integrates  the  video  data  captured  from 
the  real  world  and  the  information  from  the  UWB  radar.  ARL  conducted  an  experiment  and  demonstrated  the  real-time 
geo-registration  of  the  two  independent  data  streams.  The  integration  of  the  AR  sub-system  into  the  radar  system  is 
underway.  This  paper  presents  the  integration  of  the  AR  and  SAR  systems.  It  shows  results  that  include  the  real-time 
embedding  of  the  SAR  imagery  and  other  information  into  the  video  data  stream. 
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1.  Introduction 

The  U.S.  Army  Research  Laboratory  (ARL)  has  been  investigating  the  utility  of  ultra- wideband  (UWB)  synthetic 
aperture  radar  (SAR)  technology  for  detecting  concealed  targets  in  various  applications.  We  have  designed  and  built  a 
vehicle-based,  low-frequency  UWB  SAR  radar  for  proof-of-concept  demonstrations  in  detecting  obstacles  for 
autonomous  navigation,  detecting  concealed  targets  (mines,  etc.),  and  mapping  internal  building  structures  to  locate 
enemy  activity. 

Although  the  low-frequency  UWB  radar  technology  offers  valuable 
information  to  complement  other  sensors  due  to  its  penetration  capability,  it  is  very 
difficult  to  comprehend  the  radar  imagery  and  correlate  the  detection  list  from  the 
radar  with  the  objects  in  the  real  world.  Figure  1  shows  the  ARL  low-frequency 
radar  configured  in  forward-looking  mode.  The  radar  illuminates  the  scene  in  front 
of  the  vehicle,  and  generates  SAR  imagery  of  the  scene  and  likely  target  locations 
as  the  vehicle  moves  forward.  Figure  2a  shows  the  user’s  view  of  the  real  world 
from  inside  the  vehicle.  Without  using  any  sensor  technology,  the  user  depends 
completely  on  his  (or  her)  own  senses  to  detect  the  potential  threats  in  front  of  the 
vehicle.  On  the  other  hand,  Figure  2b  shows  an  example  of  the  displayed  SAR 
imagery  of  the  scene  that  might  help  the  user  to  identify  the  potential  threats.  The 
problem  is  that  even  though  the  information  from  the  display  of  Figure  2b  might  be  valuable,  it  is  very  difficult  for  the 
user  to  comprehend  the  radar  imagery  and  correlate  the  information  between  the  radar  image/detection  location  and  the 


Figure  1:  The  ARL  UWB  SIRE 
radar  system  integrated  on  the 
Ford  Expedition. 
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objects  of  the  real  scene.  This  conventional  user  interface  might  not  be  acceptable  for  two  reasons.  First,  the  user  needs 
to  focus  in  and  directly  interact  with  the  real  world  where  the  reaction  time  is  critical.  Second,  the  decision  based  on  both 
visible  and  SAR  information  is  very  important.  Although  the  low-frequency  UWB  radar  might  be  able  to  detect  targets 
that  are  not  visible  to  human  eyes,  it  also  detects  other  targets  that  the  user  considers  as  non-threat  targets,  such  as  trees, 
light  poles,  fire  hydrants,  etc.  Thus,  it  is  best  to  present  the  integrated  information  from  both  worlds  (real  world  and 
sensor  world)  to  the  radar  operator. 


The  emerging  augmented  reality  (AR) 
technology  addresses  this  problem.  This  technology 
combines  the  best  features  from  both  domains: 
virtual  reality  (VR)  and  the  real  world.  In  VR 
technology,  the  user  is  completely  immersed  in  the 
computer  world.  The  user  can  navigate  in  the 
virtual  world  using  information  from  the  computer 
databases  and  other  input/output  devices.  Although 

VR  is  very  useful  in  environments  such  as 
simulations  and  trainings,  in  many  other 
applications  the  user  needs  to  interact  with  the  real 
world.  AR  technology  allows  the  user  to  stay  in  the 
real  world,  but  extends  it  with  useful  information  from  the  computer  databases  and  other  sensors  to  increase  the  user’s 
perception.  In  the  simplest  form,  the  user  sees  the  world  through  a  video  camera  and  a  display.  An  AR  system  would 
overlay  the  information  from  the  computer  database  and  sensors  onto  the  live  video  stream.  The  information  from  both 
worlds  is  geo-registered  and  displayed  using  the  same  coordinate  system.  The  registration  is  performed  and  updated  in 
real-time  with  every  user’s  movement  and  perspective. 


Figure  2a:  User’s  view  of  Figure  2b:  SAR  image  of 

the  real-world  from  inside  scene  is  displayed  on  a 

the  vehicle.  comnuter  screen. 
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Figure  3:  Major  components  of  the  radar  system  and  AR  subsystem. 

Using  the  AR  technology,  we  are  exploring  the  approaches  of  incorporating  the  information  from  the  low- 
frequency  UWB  radar  sensor  into  the  live  video  display  of  the  scene  to  assist  the  radar  operator  to  determine  the  real 
threats  ahead.  Section  2  gives  an  overview  of  the  radar  system,  the  SAR  imaging  geometry,  and  the  imaging  process. 
The  SAR  imagery  generated  by  the  radar  is  completely  geo-referenced  using  the  Universal  Transverse  Mercator  (UTM) 
coordinate  system.  The  integration  of  the  AR  subsystem  and  the  radar  into  a  single  system  is  underway  and  is  described 
in  section  3.  Finally,  the  paper  shows  the  result  from  an  AR  experiment  to  demonstrate  the  real-time  registration  and 
display  of  SAR  information  onto  the  live  video  stream. 
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2.  Overview  of  the  ARL  UWB  radar  hardware  and  SAR  image  formation 

In-depth  detail  of  the  SIRE  radar  system  is  discussed  in  [1].  Figure  3  shows  the  major  components  of  the  radar. 
The  components  inside  the  dashed  rectangle  are  additional  components  from  the  AR  subsystem.  The  radar  is  a  UWB, 
time-domain,  impulse-based  system.  Down-range  resolution  is  provided  by  the  bandwidth  of  the  transmitted  monocycle 
pulse,  which  occupies  300-3000  MHz.  Sixteen  identical  Vivaldi-Notch  receiving  antennas  are  used  to  provide  a  physical 
aperture  for  the  forward-looking  mode.  Two  TEM  horn  transmitting  antennas  are  located  at  both  ends  of  the  receiving 
array.  The  ARL  UWB  radar  employs  a  stationary  GPS  receiver  (located  at  some  distance  from  the  radar)  and  a  rover 
GPS  receiver  (attached  to  the  radar  antenna  frame)  to  provide  the  radar  geo-referenced  coordinates  that  are  necessary  for 
the  image  formation  algorithm.  Both  receiver  units  are  configured  for  real-time  kinematic  (RTK)  mode.  The  stationary 
GPS  receiver  sends  the  correction  information  to  the  rover  GPS  receiver  via  a  radio  link  to  provide  an  accuracy  of  less 
than  2  cm.  The  radar  operation  is  controlled  by  a  graphical-user-interface  (GUI)-based  PC  workstation  with  high-speed 
data  acquisition  hardware  to  acquire  the  digitized  radar  data.  All  of  these  radar  components  are  packaged  onto  a  mobile 
test-bed  using  a  modified  Ford  Expedition  shown  in  Figure  1 . 


Figure  4  shows  the  radar  data  imaging 
geometry  in  the  forward-looking  mode.  The 
range  coverage  for  this  implementation  is 
designed  to  have  26  m  of  range  swath 
available,  with  a  starting  range  of  8  m  in 
front  of  the  receiving  antenna  array  and  an 
ending  range  of  34  m.  The  radar  is  a  wide 
bandwidth  impulse-based  UWB  system.  The 
downrange  resolution  is  a  function  of  the 
c 

signal  bandwidth  (A  =  — ) ,  where  c  is  the 

r  2  B 

speed  of  light  and  B  is  the  bandwidth  of  the 
radar  signal.  ARL  has  developed  a  suite  of 
signal  processing  and  image  formation 
algorithms  to  remove  noise  and  artifacts  from 
radar  data,  and  provide  an  SAR  image  with 
resolution  in  both  down-range  and  cross-range 
direction.  Figure  5  shows  the  overall  block 
diagram  of  the  signal  processing  and  image 
formation  steps.  The  PC  acquires  the  digitized 
radar  data.  Three  signal  processing  steps  (self¬ 
interference  extraction,  forward-motion 
correction,  and  frequency  sub-banding)  are  then 
applied  to  remove  noise  and  artifacts  embedded 
in  the  radar  data.  Finally,  the  image  formation 
algorithm  is  applied  to  generate  SAR  imagery. 
Detailed  descriptions  of  the  signal  processing 
and  image  formation  algorithms  are  included  in 
[2]  and  [3].  It  is  important  to  note  here  is  that 
every  pixel  in  the  resulting  SAR  image  is  geo- 
referenced  using  the  UTM  coordinate  system 
provided  by  the  two  GPS  units  operated  in  RTK 


Figure  4:  The  ARL  UWB  SIRE  radar  imaging  geometry  in 
forward-looking  mode. 
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Figure  5:  Signal  processing  and  SAR  image  formation. 

mode.  Thus,  any  information  extracted  from  the  SAR  image  (such  as  likely  target  locations)  is  also  geo-referenced. 


3.  Integration  of  the  augmented  reality  into  the  UWB  radar 

Figure  6  shows  the  basic  projection  concept  that  AR  technology  employs  to  geo-reference  objects  from  a  scene  of 
the  real  world.  The  figure  shows  a  camera  lens  that  is  located  at  the  center  of  the  geo-reference  coordinate  system.  In  this 
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figure,  the  Z-axis  of  the  coordinate  system  is  aligned 
with  the  axis  of  the  camera  lens.  A  point  object  W  from 
the  scene  with  3D  coordinates  (xw,yw,zw)  is  projected 

by  the  camera  lens  onto  a  point  /  of  coordinates 
(x7 ,  y2  ,-/ )  in  the  image  plane.  The  value  /  is  the  focal 
length  of  the  camera.  This  is  a  projection  from  the  3D 
world  onto  a  2D  plane.  The  projection  operation  can 
be  described  as 

I  =  C-W,  (l) 


where  /  and  W are  3x1  vectors,  and  C  is  a  3x3  camera  projection  matrix. 
Equation  (1)  can  also  be  written  as 


X, 

0 

o' 

xw 

yi 

= 

0 

-/ 

0 

yw 

i 

_  0 

0 

1 

_zw_ 

In  practice,  the  camera  is  not  located  at  the  center  of  the  geo-referenced  coordinate  system,  and  the  axis  of  the 
camera  is  not  aligned  with  any  axis  of  the  coordinate  system.  Equation  (1)  becomes 

I  =  C  •  R-(W  -T)  , 


lEfena  I3i  BMEQ 


(3) 

where  R  is  a  3x3  rotation  matrix 
resulting  from  the  arbitrary 
orientation  of  the  camera,  and  T  is 
a  3x1  vector  of  the  translation  of 
the  camera  lens  with  respect  to  the 
center  of  the  geo-referenced 
coordinate  system. 

Equation  (3)  shows  that  a 
projection  from  3D  world  to  2D 
image  plane  can  be  done  if  the 
coordinates,  orientation  (azimuth 
and  elevation  angles),  and 
parameters  (field  of  view,  pixel 
size)  of  the  camera  are  available. 
Figure  3  shows  the  integration  of 
the  AR  system  into  the  radar 
system.  The  major  components  of 
the  AR  systems  are  shown  inside 
the  dashed  rectangle  of  Figure  3. 
The  AR  system  receives  the 
coordinates  of  the  camera  lens 
using  the  same  GPS  system  of  the 
radar.  However,  since  the 
orientation  of  the  camera  is 
critical  for  the  geo-referenced 
projection  in  the  AR  system,  an  inertial  measurement  unit  (IMU)  is  integrated  to  provide  this  information.  The  PC 
communicates  with  the  video  camera  and  mount  to  control  the  desired  orientation  and  field  of  view. 


Figure  6:  AR  experiment  result  shows  that  SAR  information  is  overlaid 
onto  a  live  video  stream. 
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Figure  7:  SAR  image  is  overlaid  on  a  digital  Figure  8:  Radar  path  can  be  recorded  and 

photograph  of  an  area.  displayed  for  route  planning. 


Figure  6  shows  the  AR  experiment  result.  In  this 

experiment,  five  calibration  targets  (trihedrals)  are  in  the  scene.  The  radar  collects  data  and  forms  SAR  imagery  of  the 
scene.  The  geo-referenced  SAR  image  and  the  surveyed  locations  of  the  calibration  targets  serve  as  inputs  to  the  AR 
software  (from  ITT  Industries),  where  the  radar  information  is  overlaid  onto  the  live  video  stream  from  the  camera.  In 
this  experiment,  the  UTM  coordinates  of  the  targets  came  from  an  independent  surveyed  source  (not  derived  from  SAR 
image)  in  order  to  demonstrate  the  registration  capability  of  the  AR  system.  These  target  locations  are  marked  by  the 
cone  symbols.  In  practice,  these  markers  would  represent  the  likely  target  locations  that  the  detection  algorithm 
generates  using  SAR  imagery.  In  the  figure,  the  SAR  image  (formed  using  the  ground  plane)  is  overlaid  on  top  of  the 
video  of  the  scene.  The  large  responses  from  the  SAR  image  (red  color)  are  aligned  with  the  actual  calibration  targets. 

Although  the  actual  targets  are  not  clearly  visible  in  this 
figure,  the  transparency  of  the  SAR  image  can  be  adjusted  to 
allow  the  user  to  examine  both  the  video  and  the  SAR 
response  of  the  targets.  Using  the  registered  information 
from  both  video  and  SAR  images  would  help  the  user  to 
quickly  determine  if  an  object  is  likely  to  be  a  real  threat  or 
just  a  benign  object.  The  registration  is  performed  in  real¬ 
time,  and  adjusted  with  every  movement  and  perspective  of 
the  camera.  The  user  can  click  on  any  point-of-interest;  the 
camera  will  automatically  change  its  orientation  and  zoom  in 
the  area  of  interest. 

Figures  7  to  9  illustrate  other  AR  software  features. 
In  addition  to  the  overlay  of  SAR  information  and  live 
video,  the  software  can  also  import  digital  overhead 
photographs  from  computer  database.  This  allows  the  bird’s- 
eye  view  of  the  area  using  both  SAR  imagery  and  overhead 
photographs.  Figure  7  shows  a  SAR  image  that  is  overlaid 
on  a  digital  photograph  of  an  area.  Although  this  is  not  an 
AR  feature,  it  would  allow  an  image  analyst  to  examine  and 
compare  SAR  image  with  the  actual  terrain  features  of  the 
area.  Figure  8  shows  another  feature  of  the  software  that  overlays  the  path  of  the  radar  onto  the  display.  This  would  help 
the  user  to  follow  a  particular  planning  path.  In  radar  change  detection  application,  the  user  needs  to  follow  the  same 
route  as  previous  runs.  The  radar  detection  software  compares  the  SAR  image  of  the  current  run  versus  a  database  of 


Figure  9:  User  can  report  the  coordinates  of  the 
danger  areas. 
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SAR  images  from  previous  runs  in  order  to  detect  any  changes  in  the  scene.  Figure  9  shows  another  AR  software 
feature.  Once  the  user  identifies  a  suspected  area  based  on  the  information  from  the  video  and  the  SAR  information,  the 
software  allows  the  user  to  generate  a  report  by  creating  a  polygon  to  represent  the  area  using  the  mouse  pointer. 


4.  Summary 

Low-frequency  UWB  SAR  offers  the  penetration  capability  to  detect  concealed  targets.  Although  the  information 
from  SAR  offers  valuable  information  and  complements  other  sensors  that  do  not  have  the  penetration  capability,  it  is 
very  difficult  to  comprehend  the  radar  imagery  and  correlate  the  detection  list  from  the  radar  with  the  objects  in  the  real 
world.  ARL  has  been  collaborating  with  ITT  Industries  to  customize  the  AR  software  to  import  SAR  information.  ARL 
conducted  an  experiment  to  demonstrate  the  feasibility  of  using  an  AR  system  to  embed  SAR  information  onto  a  live 
video  stream  of  a  real-world  scene.  The  registration  of  the  video  stream  and  SAR  information  using  the  same  geo- 
referenced  coordinate  system  is  performed  in  real-time  and  adjusted  with  every  movement  and  perspective  of  the 
camera.  Results  from  our  experiment  showed  that  the  AR  system  successfully  performed  the  registration  with  good 
accuracy.  In  the  near  future,  ARL  will  incorporate  a  number  of  new  capabilities/features  to  enhance  the  AR  system:  a) 
provide  better  user  interface,  b)  record  geo-referenced  live  video  for  automatic  target  detection  development,  and  c) 
incorporate  an  IR  camera. 
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